feat(profiles): security-pentest, recon and scanning inside an engagement (step 5, PR 5) - #21

Phase 10, step 5 plan, PR 5 (scanning tier, T1).  Which scripts a
--script expression selects is now worked out the way nmap does it --
names, categories, globs, all, and/or/not, commas -- from nmap's
script.db, and the step is refused when any selected script is in a
category beyond scanning (exploit, brute, dos, intrusive, fuzzer) or
reaching past the command line (broadcast, external, targets-*).

This replaces matching words of the expression, which refused
"default and not broadcast"-style exclusions and could not see that
the vuln category holds exploit and dos scripts.  It also shows that
nmap's own default category holds two external scripts
(http-open-proxy, socks-open-proxy), so -sC and -A are refused with a
hint: --script "default and not external".  Script files run unchecked
code and are refused; so are names nmap does not know.  Without
script.db no script can be judged, and --script, -sC and -A are
refused.  -sV stays allowed, though nmap marks one version script,
iec61850-mms, intrusive; the code says so.

* src/codebud_sec/scope.py (_NMAP_WIDE_CATEGORIES, _NMAP_WIDE_SAMPLES):
Remove.
(_NMAP_SCRIPT_DBS, _NMAP_REFUSED_CATEGORIES, _NMAP_ENTRY)
(_nmap_scripts, _nmap_select): New.
(_nmap_script_problem): Judge the scripts selected.
(_nmap): Judge -sC and -A as default.
* tests/test_scope.py (SCRIPT_DB, script_db): New fixture.
(test_ordinary_scripts_run): -sC and default are refused now.
(test_scripts_beyond_scanning_are_refused)
(test_default_scripts_hold_external_ones)
(test_without_nmaps_script_database_no_script_is_judged): New tests.
* docs/chapters/development.texi: Describe them.

Phase 10, step 5 plan, PR 5 (T0 and T1 rules).

whois has a parser: one query, passive like a DNS lookup (out of scope
it asks); a server given with -h is judged too; an AS number names no
host.  masscan's --rate is capped at 1000 packets a second (its default
is 100), so a scan does not take the client's network down.  curl and
wget may not send a file's contents to a target -- curl -T, @file data,
form fields and headers, wget --post-file and --body-file -- so files of
the project (evidence, other engagements) never leave for a target;
data written in the step itself is fine.

* src/codebud_sec/scope.py (MASSCAN_MAX_RATE): New.
(_masscan): Refuse a faster or unreadable --rate.
(_curl_sends_a_file): New.
(_curl, _wget): Refuse sending a file.
(_WHOIS_VALUE, _WHOIS_FLAGS, _whois): New.
(PARSERS): Add whois.
* tests/test_scope.py, tests/test_scope_web.py: Test them.
* docs/chapters/architecture.texi: Mention whois.

Phase 10, step 5 plan, PR 5: tiers T0 (lookups) and T1 (scanning) of
the accepted plan (D14i); exploitation (T2) waits for the scope fence.

The profile runs no command and no web search until an engagement is
open (engagement = true); the engagement's scope guard then reads every
host each network step contacts and refuses what is out of scope.  Mode
ask; privacy private, since locked refuses network steps; encrypted;
integrity required.  Reading results is harmless; every network tool is
powerful and asked each time: dig, host, whois, nmap, masscan, nc,
ncat, curl, wget, whatweb, nikto, gobuster, ffuf, feroxbuster, sslscan,
testssl.sh.  The profile refuses listeners (nc -l, ncat --listen,
--broker, --chat), which catch shells.  Its prompt states the rules the
guard enforces, so plans meet them the first time.

* src/codebud_sec/data/profiles/security-pentest.toml: New profile.
* src/codebud_sec/data/profiles/security-pentest/: Its policy files.
* src/Makefile.am (nobase_dist_python_DATA): Install them.
* tests/test_security_pentest.py: New file.
* tests/test_profiles.py (test_builtins): List it.
* docs/chapters/configuration.texi (Profiles): Document it.
* docs/chapters/development.texi: List the test file.
* NEWS: Mention it.

Review of #21.  feroxbuster --data @file sends the file as the request
body, as curl does; it is refused like curl's.  nikto's tuning class 6
is its denial-of-service tests, outside the scanning tier: nikto now
needs -Tuning leaving it out (x6, or a list of classes without 6).
The profile's prompt and the manual say so.

* src/codebud_sec/scope.py (_feroxbuster): Refuse --data @file.
(_nikto): Require -Tuning without class 6.
* src/codebud_sec/data/profiles/security-pentest.toml (prompt): Ask for
-Tuning x6.
* tests/test_scope_web.py: Give nikto -Tuning x6.
(test_feroxbuster_data_from_a_file_is_refused)
(test_nikto_without_leaving_out_dos_is_refused)
(test_nikto_leaving_out_dos_runs): New tests.
* docs/chapters/configuration.texi (Profiles): Document them and nmap
--datadir.